Homogeneous bipartition based on multidimensional ranking

We present an algorithm which partitions a data set in two parts with equal size and experimentally nearly the same distribution measured through the likelihood of a Parzen kernel density estimator. The generation of the partition takes O( 1 2 N(N − 1)) operations (N number of data) and is 2 orders of magnitude faster than the state of the art. 1 Generating an equi-distributed bipartition 1.1 Problem and applications We consider the problem of generating a homogeneous partition of a data set, that is a partition such that each part has the same distribution as the whole. We focus here on bipartitions, which are partitions containing two parts with equal size (the number N of data is even) (Figure 1a). This problem has been studied under the name ”data squashing” [2] to summarize massive data sets in a way which preserves statistical relationships among variables better than random sampling; in biostatistics to select patients for treatment and control groups [5]; and in Machine Learning as a way to improve the estimation of model complexity [7]. 1.2 Inspiration from two-sample tests To check whether two groups are drawn from the same distribution, one can use a two-sample test. Suppose a blue-red bipartition of the data. Several two-sample tests are based on building a proximity graph of the data and counting the number of mixed edges, i.e. which have one red and one blue vertices. The higher this number, the higher the probability that both red and blue data are drawn from the same distribution. Then the null hypothesis that both parts are homogeneous is rejected if the number of mixed edges is too small. Friedman and Rafsky [3] proposed a test based on the Minimal Spanning tree. Recently, Rosenbaum proposed the ”cross-matching” test [6], which uses a minimal distance non-bipartite matching [4]. The homogeneous bipartitionning problem can bring down to build a bipartition of the data which succeeds in passing a multivariate two-sample test. One way to do this is to generate several random bipartitions and keep the one A proximity graph connects two points if they are close to each other with respect to some measure of closeness. Example of such graphs are the Minimum Spanning tree, the Nearest Neighbor graph or the Delaunay graph.